It is often desirable to capture still images of a moving scene with a camera having an electronic imaging system. Conventional electronic imaging systems include a rectangular image sensor, having an array of electronic pixels, positioned in the focal-plane of a lens to receive an optical image of the scene. Electronic pixels include a photosensitive element that transduces incident light energy into electronic potential. Most pixels operate by providing for electronic charge to accumulate for a period of time in a storage capacitance. Each storage capacitance exhibits a voltage potential that is proportional to the total light energy arriving at the pixel over the exposure time. In an electronic image sensor the exposure time must be long enough to generate pixel output voltages that are sufficiently distinguishable from the effects of noise, so as to capture an image that is a useful representation of the scene.
A distortion in the captured image, commonly known as motion-blur, occurs when there is a movement in the observed scene that forms an optical image with respect to the image sensor during exposure time. Unlike a still image where, ideally, the captured pixel values map uniquely to light emanating from points on observed objects in the scene, in a motion-blurred image, at least some pixel values are a function of the integral of the light intensities emanating from multiple points in the scene during the exposure time interval.
A very common approach to the problem of motion-blur is to shorten the exposure time and thereby marginalize the influence of motion on the captured image. This is a viable approach to the extent that the optical image possesses light energy that is sufficient to satisfy the signal-to-noise constraint noted generally above. This may require the use of impractical amounts of illumination at the scene and/or use of a large lens aperture that adversely affects the image depth of field. If the exposure time can not be shortened sufficiently, an alternative approach is to synchronize the motion of the image sensor with the motion of the imaged scene during the exposure time. This approach works well when a shared external reference frame can be used to facilitate synchronization, as with, for example, the inertial stabilization of image sensors in hand held cameras or the encoder synchronization of time-delay-integration line-scan cameras.
There are, however, applications where the optical image either does not possess sufficient light energy to allow for a short exposure time and/or the motion of the image sensor cannot be practically synchronized with the motion of the optical image. In such applications, capturing an image distorted by some amount of motion-blur (also termed herein a “blur image” or “blurred image”) cannot be avoided and a still image can only be achieved through a computational process that uses a de-blurring algorithm to extract a still image representation from the captured image data.
As described above, motion blurring results when light emanating from points on observed objects in the scene that form the optical image move with respect to the image sensor during exposure time. The pattern created in the image sensor by one such point source is known as the point-spread-function (PSF). The collection of point-spread-functions across the image define a blur transform. De-blurring algorithms estimate the blur transform, solve for the inverse blur transform, and apply the inverse blur transform to the captured image data to arrive at a still image.
Theoretically, point-spread-functions could be completely uncorrelated across the image. If this were the case, de-blurring the captured image would be virtually impossible. In practice, de-blurring algorithms make numerous simplifying assumptions. For example, it is common to assume that the point-spread-function is known and/or spatially invariant over the image or large segments of the image.
Even given such simplifications, it may be difficult or impossible to arrive at a good solution for the inverse blur transform. This is because the motion of the image relative to the image sensor over the exposure period operates as a low-pass box-filter that irretrievably destroys (or highly attenuates) significant spatial information. Although it is possible to create an approximate inverse to such a filter, the inverse, due to the fact that it is attempting to reconstruct highly attenuated or lost information, becomes very sensitive to input data, which is known as being ill-conditioned. When the solution to the inverse blur transform is ill-conditioned small changes in the assumption with regard to how the blurred image was formed can lead to errors in the resultant still image, derived from applying the transform solution, that are hugely out of proportion to their apparent magnitude in the captured image. Captured images invariably include a noise term that is independent of the blur transform, and this leads to inaccuracies in the resultant still image. For example, the still image may contain fixed pattern noise or temporal noise. Fixed pattern noise is often due to dissimilarities of transfer gate efficiency or storage capacitance between pixels in the sensor. Temporal noise often occurs due to shot noise in the photodetector or amplifier noise. De-blurring algorithms typically manage this problem by either attempting to filter the independent noise component in the captured image and/or forcing a modification on the point-spread-function that improves the conditioning of the inverse blur transform.
Most de-blurring algorithms do not assume any ability to influence the original blur transform. Such algorithms simply operate, to various extents, with a digital input image and some very general a priori expectations regarding image noise. Such algorithms are, in effect, restricted to attempting to separate the influence of noise from an ill-conditioned solution to the inverse blur transform, which is, in itself, an extremely difficult problem.
It should be noted that the art provides a class of algorithms that are capable of estimating the image motion as input, and this estimate can be employed to aid in deriving a still image by various de-blurring algorithms. For example, Ben Ezra et. al. employ a secondary motion detector to estimate image motion, and employ this estimate to aid in deriving a still image, as described, by way of background in Motion-Based Motion Deblurring, Moshe Ben-Ezra and Shree K. Nayar, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 26, NO. 6, JUNE 2004.
Other, more-recently developed, de-blurring algorithms incorporate control of the image exposure process. One such algorithm includes the use of multiple exposures, where by the total image exposure time is long enough to achieve good signal-to-noise in the captured image and the induced modification of the point-spread-function leads to an inverse blur transform that is optimal in terms of conditioning. A method for finding such a multiple exposure pattern includes evaluating various exposure patterns until the pattern that results in a sufficiently small condition number in the inverse blur transform is found. The multiple exposure process itself includes repeatedly stopping and starting the integration of light measurement over an exposure-time interval. A description of one such method is given in U.S. Published Patent Application No. 2007/0258706 A1, entitled METHOD FOR DEBLURRING IMAGES USING OPTIMIZED TEMPORAL CODING PATTERNS, by Raskar, et al., and related applications thereto, the teachings of which are expressly incorporated herein by reference as useful background information.
One disadvantage of the method described above is that for the modified point-spread-function to have a substantial influence on the conditioning of the inverse blur transform, it should incorporate numerous exposure activation periods of varying duration separated by a similar set of exposure deactivation periods. As the size of the point-spread-function grows, so do the chances that one or more simplifying assumption may be violated. For example, it is common to expect the velocity of the scene, or of objects in the scene, to be constant over the exposure period—as the duration of the period is relatively short and change may occur in a longer time than the duration of the exposure period. As the duration of the exposure time increases the robustness of a constant velocity assumption decreases. With a large point-spread-function the inverse blur transform is more influenced by boundary conditions, meaning that it is more dependent on the state of pixels that are outside the field-of-view of the captured image, as compared to a smaller point-spread-function. In addition, the accurate de-blurring of objects in the scene having a different velocity than their immediate surroundings may be restricted by the relative size of the point-spread-function, since the spatial invariance simplification will be violated on a boundary that is related in size to the point-spread-function. That is, the point-spread-function is not consistent for elements of the image that are not moving versus elements that are moving, and thus, a point-spread-function that applies to a smaller-neighborhood is desirable to reduce the effects of boundary conditions.
Another disadvantage to certain prior art deblurring algorithms, and particularly those employing intermittent starting and stopping of integration, is that the amount of light in a given exposure period is reduced. The lens aperture must generally be increased to satisfy the signal-to-noise constraints of the sensor, thereby providing sufficient light to the sensor to form an acceptable image within the integration period. Increasing the aperture can adversely affect the depth of field of the imaging system.
A system capable of capturing a readily de-blurred image of a moving scene while operating on a significantly shorter time interval, as compared to known methods, would be desirable. The ability to capture such an image employing a reduced aperture for enhanced depth of field is also desirable.